Skip to content

eval: round 15 (prep, frozen prereg, 144 runs, provisional results) - #71

Open
ryansolid wants to merge 63 commits into
mainfrom
claude/project-thread-xp2irv
Open

ryansolid wants to merge 63 commits into
mainfrom
claude/project-thread-xp2irv

Conversation

@ryansolid

@ryansolid ryansolid commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

This branch holds round 15, the six-arm, two-task round meant for publication. It covers the prep, the frozen preregistration, all 144 runs and their grades, and the provisional results.

Round 15 results (PROVISIONAL), in eval/results/round15-2026-11-16/

  • None of the preregistered hypotheses is supported (score.txt):
    • H15a, Haiku: rx < React fails, p .89. The closest comparison is rx < Exact, p .031 (.155 after Holm).
    • H15b, Opus: rx − React = +1.25, 90% interval [0.33, 2.17].
    • H15c: fails.
  • The two tasks split:
    • desk9: rx has the fewest defects, Opus 0.00 and Haiku 1.50 (React 3.17 and 9.00).
    • board (held out): rx Opus 6.33 and Haiku 25.17 (React 0.67 and 3.00).
  • Every .rx failure was traced with hold probes (trace/):
  • Open question: whether the "a write lands after the page was left" tests count, given the caching ruling. They are excluded in every arm here; score-count-left.txt counts them.
  • Also in the folder:
    • descriptives: cost, time and lines per run (descriptives.tsv; $128.74 total);
    • all 144 apps' sources;
    • PREREG-15.md: frozen on rc.23 (deviations listed), plus a record of the two container restarts and the four desks that were run again.

Prep (earlier on this branch)

  • New held-out task tasks/board: brief, API, fixtures; tests (visible 26, hidden 45, overlap 26); harness/fuzz-board.mjs (allowances B1–B4).
  • References: board references in all six arms, all grade 0 with by-design holds excluded (rx = rx-cols). desk9 references for react, xstate, foldkit and vanilla.
  • .rx changes: keys at any depth (ST-4a); guide "One store over the server"; catchRead snapshot in handlers; ARGUMENT_COUNT; props in C() UNBOUND; DV-22 extended.
  • Fuzz rules: --r15 drops A2. A4 now covers writes that land after their page was left. The repeated-key answer is the first one the server accepts. B4 is new.
  • Solid repros for earlier bugs: in results/r15prep/repro/.

🤖 Generated with Claude Code

https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
A write that lands after its page was left is not judged on the next visit
(Ryan 2026-11-14: late writes are out of scope). A4 was only applied to rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…XState references follow it

Ryan 2026-11-14: with two accepted writes to one field, the server's last answer is the
only correct value at rest. --r15 drops allowance A2; FUZZ_ROUND=--r15 sh harness/grade.sh.
All six desk9 references grade 0 under it (20 seeds; React, Exact, XState, Foldkit and
vanilla also 0/60 on seeds 21-80).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…erence, fuzz seed 45)

Also replay.mjs: DW=<n> widens the detail text it prints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…ence (grades 0)

Brief frozen 2026-11-14 (Ryan). 26 visible, 45 hidden, 26 overlap tests; fuzz-board.mjs
with oracles for duplicates, headings, counts, the work-in-progress limit, the detail,
and settled state. TASK=board sh harness/grade.sh grades it. The fuzzer catches two
planted bugs in the reference (limit off by one, an accepted toggle not kept).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…el under a pending guess

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…rite visit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…15 prereg draft

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Failed ❌

View logs ↗
55f0320 2026-11-17T21:16:46.550Z View logs ↗
  • Build: Failed ❌

View logs ↗
c854278 2026-11-17T18:49:04.650Z View logs ↗
  • Build: Failed ❌

View logs ↗
aac0bc3 2026-11-17T16:58:13.543Z View logs ↗
  • Build: Failed ❌

View logs ↗
52a0997 2026-11-16T19:55:12.222Z View logs ↗
  • Build: Failed ❌

View logs ↗
3289544 2026-11-16T11:29:18.797Z View logs ↗
  • Build: Failed ❌

View logs ↗
192e36c 2026-11-16T11:06:38.460Z View logs ↗
  • Build: Failed ❌

View logs ↗
cd47f41 2026-11-16T10:57:29.043Z View logs ↗
  • Build: Failed ❌

View logs ↗
2d357af 2026-11-16T10:37:37.044Z View logs ↗
  • Build: Failed ❌

View logs ↗
670f26e 2026-11-16T07:00:06.468Z View logs ↗
  • Build: Failed ❌

View logs ↗
486af79 2026-11-15T22:42:14.191Z View logs ↗

View all previews: View all previews ↗

… answered add's placeholder while work is in the air

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…, minimal trail)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…s/solid#4874, rx runtime catchRead, pending probe)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…); guide: one store over the server

- Runtime: `catchRead` with no observer (a component fn called from a handler or
  action) is a snapshot (04 R-8): the committed value, or the alternative on a
  matching failure. A fresh error boundary there had no value while a refresh
  was out and threw NotReadyError (S-catch-snapshot).
- Checker (ref/ and crate): ARGUMENT_COUNT (06 AC-1f) for a call of a declared
  action with the wrong number of arguments; `props` in `component C()` is
  UNBOUND (03 RG-4b). AST gains `noProps`. S-check-1 rejected variants, a
  check.rs unit test, guide/diagnostics.md sections.
- pending? inside a held `loading on(sel)` region reads false: Solid's, under
  05 DV-22 (plain-Solid repro saved). Guide and GUESS_WHILE_LOADING text say
  to test `pending? x` in the handler too (R-12).
- guide/async.md: "One store over the server".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…k shown column), fuzz B3 declined count hold, repro for solidjs/solid#4887

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…on, two Solid repros

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…and by key; record seed)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…(seed 28); repro for solidjs/solid#4897

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…cile; DV-19/20/23 fixed (#64)

- ref/SOLID_NEXT -> rc.22 (221b5f9); runtime and eval deps ^2.0.0-rc.22
- landDraft returns a keyed answer from the derive (Solid reconciles by key at any depth, ST-4a);
  unkeyed answers still assigned. Drops the hand-written keyed merge from #63.
- S-guess-reask-refresh, S-latest-reask, S-solid-46: divergence records removed (fixed in rc.22)
- S-solid-44: only DV-22 (pending? under a new question) still recorded, instant 3
- All ref runners green: oracle/compiled 131/131, ssr 124 (7 skip), types 177, crate 367/367

Closes #64

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
@socket-security

socket-security Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updatednpm/​@​solidjs/​web@​2.0.0-rc.21 ⏵ 2.0.0-rc.23100 +110083 +197 +1100
Updatednpm/​solid-js@​2.0.0-rc.21 ⏵ 2.0.0-rc.23100 +110095 +196 +1100
Updatednpm/​@​solidjs/​compiler@​2.0.0-rc.21 ⏵ 2.0.0-rc.2399 +1100100 +196 +1100
Updatednpm/​@​solidjs/​signals@​2.0.0-rc.21 ⏵ 2.0.0-rc.23100 +1100100 +198 +1100

View full report

…path guess when the row leaves and returns)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
… fuzz trails (jsdom)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…ter retry; regression from #4908)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…d key; repro for solidjs/solid#4928

A key asked twice (an archive asked on two visits) was answered twice and the last answer went to the app: the
first archived the card, the second found nothing and failed, so the app reverted while the server had archived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…120)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…tem toggled twice in a busy period

From Ryan's ruling on two accepted writes to one field (settled = the server's last answer): the answers can come
back in another order than the clicks, and the screen keeps the last click until its re-ask lands. Board seed 70.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…val to rc.23

Ryan started round 15 on rc.23 (2026-11-16) and waived the gate rerun and outside review before the runs; both are
recorded as deviations under Freeze.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…15 run order

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…und while writing the references

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…69-#4971)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
@ryansolid ryansolid changed the title eval: round 15 prep (board task, six-arm references, r15 fuzz rules, prereg draft) eval: round 15 (prep, frozen prereg, 144 runs, provisional results) Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
… server's

A write into a source with a question after an `await`, whose value (closures
included) names a row parameter or a local read from a source before that
`await`, is STALE_CAPTURE. Principle: a write must not read stale state.
Round 15 board: fires on 9 of 12 .rx board apps, each on the traced defect
(stale card, wrong column, stale counts); 0 on the 19 .rx references, the
examples, the round 15 desk9 apps and 161 earlier .rx apps. ref/ and the
crate agree on all 211 app folders and on crate.ts (370/370).

Guide (async.md, three layers): two general rules, "after the await, write
only what is the server's" and "a guess and the answer that ends it share a
place". Spec 06 AC-1g (taken by default), S-infer-7, diagnostics page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…board probe

A 3-desk Opus board probe on the first AC-1g draft (pre-await reads only)
showed 2 of 3 agents moving the read after the await and copying a row
held by another action into the target column (lost rename). AC-1g now
flags any source/equation read in a post-await write into an answer, with
the same zero hits elsewhere in the corpus (3 October apps writing
x := x.map(...) after an await also hit). Guide bullet and diagnostics page
updated; S-infer-7 gains the after-await variant. ref/ and crate agree
(crate.ts 371/371). Probe results in eval/results/board-probe-2026-11-17.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
… ledgers)

hidden 77/77, overlap 38/38, fuzz --r15 1/20 (seed 10: a reopened page's list
held by a close in flight, H-11). Rename/assign are path guesses at an index
the handler passes; a row not on screen when guessed gets the answer as a
guess after the await, until the re-ask lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…time

It was not lowered and threw at run time, inside the action's catch, so the
write looked like a declined one (found rewriting the desk9 reference with
guesses on the answers: 11/20 fuzz seeds). No eval reference or round 15 app
hits it; crate.ts --lower unchanged (190/213), compiled scenarios unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
rx-one: a row close out holds the tab switch itself (the Help page stays until
the close answers; H-11, consistent, settles right). The ledger reference and
round 15 desk9-rx-opus-1 switch at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…retry as an action

Tried the two-layer board shape too (answer source + store over it, accepted
renames/assigns written into the answer, no re-ask): seed 3 lost an assign to
a list re-ask asked before it was accepted. Kept: accept = re-ask, with the
server's answer guessed on a row that wasn't on screen. Grades 77/77, 38/38,
fuzz 1/20 (seed 10, the H-11 tab-switch hold).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Replaces 'Writing one row', 'Guessing data that depends on a selection',
'Every view of a write: lay it over the answer', 'three layers' and 'One store
over the server' with 'Guesses go on the answer' (guess on every store the
write shows in; accepted = re-ask; a row not on screen gets the answer as a
guess after the await) and 'When a guess holds'. Ledgers (writes in lists of
their own laid over the answers) are no longer taught (Ryan, 2026-11-17).
The desk9 reference in this shape is eval/reference/desk9/rx-one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…he one latest says

latest of a list that hasn't answered throws, and the action's catch took it as a decline
(board probe 2, opus-1 seeds 2, 9, 18). Plain-Solid repros for solidjs/solid#4987, #4988.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
Probe 2: 7, 5, 3 defects, all apps on the shape. Probe 3: 9, 8, 12; the earlier-move defect is gone, holds are
longer, and an open detail-tear family grows with the extra re-asks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants